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ABSTRACT:  This  project  demonstrates  an  approach  to  predicting  and  optimizing  the  impact  of  new  technologies  in 
system  re-design  by  using  simulation  to  model  operator-system  functionality.  A  task  network  model  was  developed  to 
create  a  real-  time  simulation  of  the  tasks  performed  by  sonar  operators  in  building  the  underwater  picture.  This 
picture  is  created  by  analysing  sonar  data,  and  the  process  is  made  complex  by  high  volumes  of  noise  and  multiple  data 
that  arrive  from  a  variety  of  acoustic  sources,  detected  at  great  distances  by  modern,  sonar  equipment.  The  task  is 
made  difficult  by  the  fact  that  single  acoustic  sources  have  a  complex  spectrum  consisting  of  several  base  frequency 
components  and  related  harmonics.  The  task  for  operators  is  to  analyse  the  data  to  determine  if  there  is  a  pattern  that 
represents  the  signature  of  a  known  source,  thereby  leading  to  identification  of  a  vessel.  Since  the  task  can  be  highly 
labour  intensive,  automated  decision  aids  may  be  of  value  to  the  operator,  but  their  effects  on  performance  and  design 
trade-off  decisions  are  not  easily  predicted  or  intuitively  obvious.  The  task  network  model  provided  a  means  for 
developing  a  baseline  system,  against  which  the  performance  advantages  of  various  decision  aids  could  be  evaluated. 
The  specific  improvement  in  performance  predicted  by  the  model  for  one  promising  aid  was  then  validated 
experimentally. 


1.  Introduction 

This  paper  addresses  the  issue  of  how  to  evaluate  the 
effectiveness  of  new  tools  to  assist  sonar  operators  whose 
task  is  to  build  an  underwater  picture  to  guide  tactical 
decisions  in  the  Canadian  Navy.  The  underwater  picture 
contains  information  related  to  the  identity,  position, 
course  and  speed  of  surface  and  subsurface  contacts 
detected  by  acoustic  sensors.  Although  this  domain  has 
provided  the  direction  for  the  project,  the  methods  used 
and  results  obtained  generalize  to  more  generic 
environments  that  involve  the  processing  of  large 
volumes  of  complex  information  under  conditions  high 
noise  and  uncertainty. 


The  specific  goal  of  the  project1  has  been  to  assess  the 
effectiveness  of  operator  aids  to  enhance  the  process  of 
identifying  vessel  signatures  from  sonar  patterns,  using  a 
task  network  model  of  the  sonar  domain.  A  simulation 
approach  to  finding  practical  and  effective  solutions  to 
improving  human-system  effectiveness  was  chosen  for  a 
variety  of  reasons.  First,  the  operational  system  is 
complex  and  not  readily  adaptable  to  “bench  testing”  new 
software  and  hardware  components  for  evaluation 


1  This  project  was  funded  by  a  contract  to  Humansystems 
Incorporated®  by  DRDC  Atlantic  and  DRDC  Toronto 
and  the  current  paper  is  based  upon  the  final  report  by 
Matthews,  Bos  and  Webb  (2003). 
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purposes.  Second,  operational  systems  are  normally 
deployed  and  not  available  for  R&D  purposes.  Third, 
trained  operators,  who  might  be  used  to  evaluate  system 
improvements,  are  few  in  number  and  also  not  readily 
available.  The  simulation  approach  allows  for  the 
building  of  a  baseline  model  that  replicates  the  essential 
components  of  the  operational  environment,  but  then 

Figure  1 :  Representative  frequency/time/intensity  display  depicting  sonar  lines 


allows  a  variety  of  “what-if  ’  questions  to  be  addressed 
and  provides  quantitative  estimates  of  changes  in  system 
performance  and  operator  workload  that  might  be 
anticipated  by  design  enhancements. 


1.1  The  task  domain 

Sonar  data  received  by  acoustic  sensors,  often  at  great 
distances,  arises  from  a  variety  of  biological  and 
mechanical  sources  and  are  affected  in  transmission  by  a 
variety  of  environmental  and  oceanic  factors.  As  a  result, 
the  pattern  of  sonar  data  received  from  a  large  vessel  may 
contain  acoustic  frequencies  associated  with  many 
sources  such  as  engines,  shafts,  propellers,  generators, 
pumps,  switches  and  other  electromechanical  devices. 
Each  source  will  likely  generate  not  only  a  base  frequency 
(e.g.  60Hz)  but  also  harmonic  components  at  higher 
frequencies. 

The  task  for  the  sonar  operator  is  to  distinguish  among 
background  noise  the  patterns  of  frequencies  that 
represent  likely  sources  and  then  to  analyse  these  patterns 
to  determine  if  they  match  the  signature  profile  of  known 
vessels.  Information  is  typically  provided  to  operators  in 
the  form  of  a  frequency  (x  axis)  by  time  (y  axis)  by 
intensity  (z  axis)  display  that  updates  at  regular  intervals 
with  the  most  frequent  data  displayed  at  the  bottom. 
Therefore,  as  the  display  updates,  there  will  be  an 
appearance  of  an  upward  waterfall  effect.  The  display 
normally  has  a  background  level  of  “noise”  that 
represents  random  signals  arising  from  the  underwater 
environment  and  detected  by  the  system.  Signals  from 
sonar  sources  will  appear  on  the  display  as  vertical  lines 


whose  length  corresponds  to  their  duration  and  luminance 
to  their  signal  strength. 

A  sample,  representative  display  is  shown  in  figure  1 .  As 
can  be  seen,  such  lines  are  readily  detectable  from  the 
background  noise,  unless  they  are  very  brief  in  duration. 
The  signature  of  a  single  vessel  may  in  fact  comprise 
anywhere  between  25  and  100  lines  depending  upon  the 
number  of  acoustic  sources  active,  the  distance  to  the 
sensor  and  a  variety  of  oceanic  variables.  Further,  the 
signature  may  not  be  the  same  fixed  pattern,  but  oceanic 
conditions  may  cause  the  base  frequency  to  appear  shifted 
to  a  different  frequency  and  may  also  influence  the  ratio 
of  the  intensity  of  the  harmonics  to  the  base  frequency. 

Thus,  for  the  most  part,  there  is  no  simple  unique  visual 
pattern  that  represents  a  vessel  and  hence  a  visual 
identification  of  the  pattern  alone  is  not  feasible.  This  is 
particularly  the  case  when  several  vessels  are  picked  up 
by  sensors  and  their  overlapping  patterns  are  present  on 
the  display.  Thus,  the  task  is  one  of  serial  analysis  of  the 
lines  aided  by  a  set  of  tools,  such  as  a  variable  harmonic 
cursor  that  allows  lines  that  are  harmonically  related  to  be 
determined  more  readily. 

Complicating  the  process  further  is  the  fact  that  no  single 
display  can  represent  360  degrees  of  coverage  of  the 
ocean.  Therefore,  individual  sectors,  or  beams,  (radiating 
from  the  ship  responsible  for  the  picture  building)  of  the 
environment  are  filtered  and  each  allocated  to  a  given 


window  that  can  be  brought  up  on  the  display.  Typically, 
the  ocean  may  be  divided  into  anywhere  from  20  to  100 
such  beams,  although  for  present  purposes  we  have 
assumed  full  coverage  with  44  beams. 

Depending  upon  transmission  conditions,  distance  away, 
the  power  of  the  source,  and  the  overlap  of  the  beam 
response  patterns,  a  single  vessel’s  signature  may  be 
present  on  a  number  of  adjacent  beams.  Thus,  in  order  to 
build  a  complete  picture  of  the  underwater  environment 
an  operator  must  successively  step  through  and  analyse 
each  window  associated  with  a  beam,  a  process  that  may 
take  hours  under  many  operational  circumstances. 

A  further  complication  is  that  the  ship  building  the  picture 
normally  travels  in  a  task  group  (TG)  comprising  five  or 
six  vessels  in  reasonably  close  proximity.  Each  of  these 
vessels  will  generate  its  own  acoustic  pattern  that  will 
also  “flood”  the  operator’s  display  with  lines  of  data. 
Under  some  circumstances  these  lines  can  number  in  the 
thousands.  Further,  the  pattern  of  these  lines  will  change 
over  time  depending  upon  the  geospatial  relationship 
between  the  vessels  emitting  the  sounds  and  the  vessel 
doing  the  detection,  their  speeds  as  well  as  the  intervening 
oceanographic  conditions. 

Essentially,  there  are  two  critical  picture  compilation 
tasks  to  be  done.  The  primary  task  is  to  build  the 
underwater  picture  by  identifying  vessels  of  interest  and 
their  associated  signatures.  As  part  of  this  primary  task, 
each  line  detected  on  a  beam  must  be  identified  as 
belonging  to  one  of  three  categories:  “known”  -  a  part  of  a 
target  signature  that  is  unambiguously  recognised  based 
upon  information  held  in  a  knowledge  database; 
“ unknown ”  -  a  line  that  cannot  be  definitely  associated 
with  any  known  signature  and  “ possible ”  -  a  line  for 
which  there  is  some,  but  inconclusive  evidence,  that  it 
belongs  to  a  known  signature.  The  operator  is  required  to 
tag  each  and  every  line  on  the  current  beam  into  one  of 
these  three  categories,  enter  the  information  into  a  log  and 
then  move  on  to  examine  the  data  from  the  next  beam. 
For  simplicity,  we  will  refer  to  this  task  subsequently  as 
search/id. 

The  secondary  critical  task  is  to  identify  and  eliminate 
from  further  analysis  the  known  vessel  signatures  arising 
from  the  TG,  thereby  enhancing  the  ability  to  do  the  first 
task  and  the  efficiency  with  which  it  can  be  done.  This 
process  of  elimination  is  often  referred  to  as  “ sanitizing ” 
the  display. 

The  sanitization  task  is  essentially  a  top-down  process, 
whereby  the  operator  uses  known  information  about 
target  signatures  of  TG  ships  to  direct  the  search  for 
locating  on  which  beams  the  individual  lines  can  be 
found.  The  process  ends  with  the  operator  entering  into 
the  log  the  locations  and  line  components  of  all  of  the 
updated  signatures.  In  reality,  the  way  this  process  tends 
to  work  is  that  the  operator  looks  for  evidence  of  the 


individual  sound  sources  that  comprise  each  vessel’s 
signature,  such  as  acoustic  lines  associated  with  engines, 
shafts,  propellers,  generators  and  other  electro-mechanical 
devices.  Once  all  of  the  individual  sources  are  found  the 
vessel  as  a  whole,  and  all  of  its  associated  lines,  are  then 
fully  identified. 

Because  of  the  continuous  and  high  work  load  demands 
associated  with  each  of  these  tasks,  in  a  typical 
operational  environment,  one  operator  will  be  assigned  to 
build  the  picture  (search/id)  and  a  second  to  sanitize  the 
display.  When  the  latter  operator  has  finished  this  process, 
she/he  is  then  available  to  help  out  the  operator  building 
the  picture. 

Under  some  operational  conditions,  it  would  not  be 
unusual  for  the  sanitization  process  to  take  hours  to 
complete,  and  could  require  the  almost  constant  attention 
of  one  operator.  The  process  itself  does  not  place  heavy 
intellectual  demands  on  the  operator,  but  is  often  referred 
to  as  “brain  dead”  and  is  known  to  lead  to  boredom  and 
data  entry  errors,  particularly  with  extended  time  on  task. 

This  is  a  somewhat  inefficient  use  of  personnel  resources, 
and  hence  the  motivation  for  the  present  project  was  to 
identify  possible  operator  aids  that  could  assist  in  the 
sanitization  process,  thereby  freeing  up  operator  resources 
to  deal  with  the  more  tactically  critical  task  of  underwater 
picture  building. 

Further,  the  baseline  model  was  also  seen  as  a  way  of 
addressing  the  potential  operational  impact  of  future 
system  re-design  options,  including  issues  such  as: 
increasing  the  beam  resolution  and  number  of  beams, 
changing  the  number  of  displays  and  their  size,  color 
coding  of  information,  re-assignment  of  operator  tasks 
and  personnel  redeployment. 

2.  METHOD 

2.1  Building  the  model 

Using  existing  task  analyses  of  navy  sonar  systems 
(Matthews,  Greenley  and  Webb,  1991)  and  with  the 
assistance  of  a  subject  matter  expert  who  was  an 
experienced  Navy  sonar  trainer,  critical  tasks  were 
identified  that  comprise  the  processes  of  the  detection  and 
identification  of  ships  from  their  radiated  acoustic 
patterns.  These  tasks  then  formed  the  basis  for  creating 
decision-action  diagrams  to  represent  the  sequential 
operations  performed  and  decisions  made. 

For  each  task,  estimates  were  generated  of  the  time  to 
complete  (means  and  variances),  probability  of  success, 
consequences  of,  and  tasks  influenced  by,  failure  and 
operator  workload  on  visual,  auditory,  cognitive  and 
psychomotor  dimensions  (VACP).  VACP  is  an 
attentional  demand  algorithm  based  upon  the  task  loading 
for  an  operator  within  the  four  separate  channels  and 
estimates  the  demands  on  human  processing  resources.  To 
achieve  a  VACP  rating,  each  operator  task  is  rated  with 


respect  to  the  weighted  task  demand  that  appears 
appropriate  for  the  specific  task  requirements  for  each  of 
the  four  independent  channels.  Scales  to  assist  the 
generation  of  these  ratings  were  developed  originally  for 
an  LHX  mission  function  analysis  performed  by  Aldrich 
and  others  (1984),  for  the  US  Army  Research  Institute. 
The  scales  provide  a  subjective  rating  for  various  levels  of 
attentional  demand.  Additional  work  was  later  published 
by  Bierbaum,  Szabo,  and  Aldrich  (1987)  and  provided 
enhanced  descriptors  and  interval  scale  values. 

Tasks  were  assigned  workload  ratings  on  a  seven  point 
scale  by  comparing  them  to  normative  values  of  the  IPME 
workload  scale,  shown  in  full  in  Annex  A. 

2.2  Parameters  of  the  model 

The  model  simulated  a  two-operator  environment  in 
which  one  operator  detected  sonar  lines  of  interest, 
analysed  them  and  attempted  to  make  identification  from 
the  observed  pattern,  while  the  other  operator  sanitized 
the  array.  Both  operators  entered  their  analysis  of  each 
line  into  a  handwritten  log  that  contained  a  number  of 
fields  of  information  relevant  to  the  properties  of  the  line. 

2.3  System  Hardware 

In  order  to  approximate  the  realities  of  existing  sonar 
systems  the  hardware  constraints  were  set  as  follows: 

•  Two  high-resolution  monitors 

•  Processed,  narrow  band,  sonar  data  were 
represented  on  44  beams  (although  in  principle 
this  number  may  be  readily  manipulated). 

•  Data  were  represented  as  frequency  information 
over  time.  There  were  three  display  resolutions 
per  monitor  representing  single,  triple  beam  and 
search  summary  (all  44  beams)  formats. 

•  Aural  presentation  of  sonar  data  was  available  to 
operators  to  enable  further  analysis 

2.4  The  underwater  model  and  sonar  contacts 

The  detailed  simulation  of  the  complexity  of  the 
underwater  environment  and  its  interactions  with  the  wide 
range  of  sonar  data  that  are  created  by  mechanical  and 
non-mechanical  sources  was  beyond  the  scope  of  a 
baseline  model.  Instead,  the  starting  assumption  was  that 
a  number  of  sonar  sources,  each  of  which  has  a  variety  of 
sound  frequency  components  that  arrive  at  the  sensor,  are 
presented  on  a  display,  or  can  be  heard  through 
headphones  or  speakers.  These  sonar  data  may  come 
from  biological  or  mechanical  sources  whose  frequency 
characteristics  may  be  known  or  unknown  and  are 
associated  with  a  certain  probability  of  being  detected 
against  random,  background  noise.  Thus,  the  sonar 
database  comprised  a  number  of  signal  representations 
that  corresponded  to  sources  that,  when  processed  by  the 
operators,  should  result  in  identifications  of  non¬ 
mechanical,  unknown,  possible,  or  known.  The  particular 


frequency  characteristics  and  the  numbers  and  types  of 
signal  sources  are  described  below. 

2.5  Target  spatial  and  temporal  dynamics 

To  simplify  the  simulation,  the  baseline  model  did  not 
represent  the  complexities  of  TG  movement  through  the 
ocean,  therefore  sonar  sources  other  than  from  the  TG 
were  represented  on  a  single  beam  only.  The  model 
function  assigned  the  lines  of  sonar  data  randomly  to  the 
44  beams.  While  this  may  be  unrealistic  of  many 
operational  conditions  it  does  faithfully  represent  the  task 
of  detecting  and  identifying  sonar  contacts  that  are 
represented  on  a  single  beam. 

In  order  to  simulate  the  temporal  parameters  of  acoustic 
data,  the  data  representations  of  the  source  targets  were 
defined  as  having  a  finite,  temporal  lifespan  that  entered 
into  the  simulated  underwater  environment  at  varying 
points  in  time.  In  this  way,  the  information  available  to 
the  operator  changed  over  time  and,  if  the  data  were  not 
processed  before  they  expired,  then  contacts  would  be 
missed  or  misclassified.  Once  being  available  for 
processing,  each  target  had  a  life  span  of  between  200  and 
600  seconds  that  was  randomly  assigned  by  a  model 
function. 

In  order  to  represent  the  state  of  the  system  at  a  watch 
start,  during  the  first  300  seconds  of  the  simulation,  the 
array  was  populated  with  lines  and  the  search/id  and 
sanitization  tasks  did  not  commence  until  this  was 
completed. 

2.6  Sonar  data  types 

Four  characteristics  of  sonar  data  were  modelled  as 
follows;  their  probability  of  occurrence  is  shown  in 
parentheses: 

•  Source  is  a  true  target  (.22) 

•  Source  is  noise  (.22) 

•  Sonar  data  require  the  operator  to  wait  for 
additional  screen  updates  (i.e.  the  signal  is  too 
brief  to  allow  analysis  to  occur)  (.33) 

•  Sonar  data  scroll  off  the  display  before  the 
operator  has  time  to  make  an  identification.  (.22) 

Target  identification  characteristics 

This  task  could  be  modelled  in  a  variety  of  methods 
depending  on  the  level  of  complexity  of  the  human 
processes  that  are  to  be  simulated.  To  capture  the  essence 
of  the  task,  and  based  upon  Subject  Matter  Expert  (SME) 
input  as  to  what  would  be  representative,  20  data  lines  on 
each  beam  (associated  with  a  single  target)  were  required 
to  be  present  before  the  target  could  be  identified.  If  such 
lines  were  not  present,  then  the  target  would  either  be 
missed  or  classified  as  unknown.  This  approach  was 
chosen  to  generate  identification  times  that  were 


consistent  with  the  wide  range  of  actual  identification 
latencies  that  occur  under  operational  conditions. 

2.6  Taskgroup  Data 

These  data  result  from  the  sensor  array  picking  up,  on  a 
continuous  basis,  acoustic  signals  generated  by  noise 
sources  on  all  five  ships  that  comprised  the  TG.  The 
actual  number  of  lines  that  might  be  generated  by  the 
entire  TG  under  operational  conditions  could  range  from 
the  hundreds  to  the  thousands.  Based  upon  SME  input, 
two  values  were  chosen  for  the  numbers  of  lines  per  ship 
to  be  processed,  representing  a  low  and  moderately  high 
TG  “noise”.  For  the  low  load  condition,  there  were  25 
lines  per  ship  and  each  ship  was  represented  on  five 
different  beams;  therefore,  for  the  entire  five  ship  TG 
there  was  a  total  of  625  lines.  For  the  high  condition 
there  were  100  lines  per  ship,  for  a  total  of  2500  TG  wide. 


As  described  above,  there  were  two  functionally  separate 
elements  of  the  operator  model  -  the  basic  search  and 
analysis  for  contacts  of  interest  and  the  sanitization  of  the 
array  of  the  known  lines  arising  from  the  TG.  Both 
operators  conducted  a  basic  search/id  and  one  operator 


2.7  The  operator  model 

Figure  2\  Schematic  representation  of  template  assistance  functionality. 
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was  additionally  assigned  to  sanitize  the  array  at  regular 
intervals  throughout  the  watch.  In  the  standard  search/id 
process  the  operator  searched  through  the  beams  to  detect 
sonar  signals  of  interest,  identify  the  source  and  log  the 
relvant  data  for  each  line  detected.  One  operator 
sequentially  search  up  the  beams  from  1  -44  and  the  other 
from  44-1.  When  sonar  target  signals  were  encountered 
by  an  operator,  the  search  process  was  halted  and  the 
identification  process  was  started  by  that  operator  on  that 
beam.  The  operator  resumed  the  search  at  the  interrupted 
point,  once  the  identification  task  was  completed.  Thus, 
the  tasks  of  search  and  identification  could  not  be 
performed  in  parallel  by  a  single  operator  and  required 
time-sharing. 


When  one  operator  interrupted  the  search  to  analyse  data, 
the  other  operator  continued  to  search  until,  or  if,  the 
conditions  arose  that  required  this  operator  also  to  engage 
in  the  analysis  process  that  results  in  identification. 

At  the  start  of  the  watch  (i.e.  when  the  simulation 
commenced)  one  operator  " sanitized  the  array”  while  the 
other  operator  searched.  Once  this  sanitization  process 
was  complete,  the  operator  also  searched  for  contacts. 
The  sanitisation  process  was  required  to  recur  several 
times  during  the  simulated  watch,  since,  as  under  normal 
operational  conditions  TG  lines  would  migrate  across 
different  beams  due  to  the  changes  in  relative  positions  of 
the  sensors  and  the  various  ships.  The  recurrence  interval 


for  the  sanitization  procedure  was  set  at  20  minutes,  based 
upon  SME  input  for  the  conditions  that  were  being 
simulated. 

2.8  Running  and  debugging  the  model. 

As  the  model  ran,  variables  were  displayed  to  monitor 
their  values,  and  the  event  queue  was  also  monitored  for 
events  waiting  to  be  executed.  The  trace  file  was  also 
enabled  to  record  the  time  when  each  task  began  and 
ended.  These  options  assisted  in  verifying  that  the  model 
was  operating  as  intended,  and  identified  where  changes 
were  needed.  Once  the  model  was  running  smoothly, 
snapshots  were  defined  to  collect  values  of  variables  at 
specified  points  during  model  execution.  These  provided 
further  validation  that  the  model  was  operating  correctly 
and  identified  possible  problem  areas. 

2.9  Modeling  operator  tools. 

Two  labor-intensive,  error-prone  tasks  of  the  operational 
environment  were  considered  to  be  prime  candidates  for 
automation  or  operator  assistance,  namely  the  sanitization 
process  and  the  logging  of  data.  The  decision  was  made 
to  concentrate  initially  on  the  former  process  by 
considering  the  kind  of  tool  that  would  help  the  operator 
to  perform  this  process  more  efficiently. 

To  review,  the  current  sanitization  process  is  a  top-down 
serial  search  for  expected  acoustic  sources  that  comprise 
the  signature  lines  of  each  TG  ship  on  expected  beams 
followed  by  an  update  of  the  log  to  reflect  the  actual 
current  line  data.  This  process  is  repeated  for  all  ships 
until  all  their  lines  are  accounted  for.  Obviously,  this  task 
involves  a  lot  of  back  and  forth  checking  between  the  log 
and  the  display,  and  requires  the  mental  translation  of 
written  frequency  values  and  beam  numbers  in  the  log 
into  where  to  look,  and  what  to  look  for,  on  the  actual 
display.  It  seems  feasible  that  a  tool  could  be  created  that 
would  use  the  information  in  the  log  to  create  a  visual 
template,  or  pattern  of  lines,  that  when  overlaid  on  the 
display  would  provide  instantaneous  feedback  as  to 
whether  those  lines  were  actually  present  on  the  beam. 

Figure  2  depicts  the  functional  elements  and  the  process 
sequence  of  how  the  template  assistance  works.  Data 
accumulated  on  the  signature  history  of  each  ship  in  the 
task  group  is  fed  into  a  database.  This  in  turn  is 
integrated  with  concurrent  information  on  the  locations  of 
ships  in  the  TG,  their  speed  and  current  oceanographic 
conditions  using  an  artificial  intelligence  (AI)  engine. 
The  outcome  of  this  process  is  an  identification 
hypothesis  that  predicts  each  vessel’s  signature 
components  and  their  beam  location  relative  to  the 
sensing  source.  This  information  is  then  translated  into  a 
visual  template  to  overlay  the  beam(s)  on  which  the 
signature  is  expected  to  be  found.  The  nature  of  the 
overlay,  as  shown  in  Figure  3  below,  is  that  the  template 


pattern  comprising  25  lines  amplifies  the  luminance  of 
the  underlying  sonar  data  when  they  are  co-located, 
thereby  making  each  signature  line  to  appear  brighter  than 
other  data  lines.  Also,  the  template  extends  a  few  pixels 
above  and  below  the  data  area  to  facilitate  the  location  of 
the  template  lines. 


Figure  3:  Sonar  display  with  ID  template  overlaid 


The  operator  uses  this  template  to  make  a  judgment  of 
whether  the  signature  is  present  or  not  and,  if  present,  a 
smart  data  entry  process  captures  the  required 
information  for  the  log.  If  the  signature  is  not  found,  the 
operator  analyses  the  line  pattern  found  and  then  updates 
the  database  if  the  pattern  is  found  to  be  at  variance  from 
the  predicted  identity. 

Thus,  from  the  operator’s  perspective,  the  visual  analysis 
component  of  the  sanitization  task  is  greatly  simplified 
and  the  human  proficiency  in  visual  pattern  matching  of 
complex  features  is  capitalized  upon. 

The  feedback  from  the  Navy  sonar  SME  on  the  proposed 
semi-auto  sanitisation  process  was  very  positive  and  the 
functionality  was  very  much  in  line  with  what  he 
envisaged  would  be  an  optimum  approach. 

Accordingly,  a  process  model  was  developed  to  analyse 
and  simulate  how  such  a  template  aid  might  work,  the 
model  was  then  run  and  debugged. 

2.10  Model  execution 

Once  the  overall  model  was  found  to  be  error  free  it  was 
run  with  30000  data  updates  generated  at  a  rate  of  1  per 
second,  thereby  simulating  8.3  hours  of  real  time 
operating  conditions,  during  which  time  approximately 
three  thousand  lines  of  sonar  data  were  entered  for 
analysis.  Ten  model  runs  were  executed  for  each  of  the 
conditions  in  which  the  sanitization  aid  was  available 
(assisted  condition)  or  not  (baseline  condition).  Further, 
two  conditions  of  TG  load  were  simulated,  in  which  each 
TG  ship  was  represented  by  either  25  or  100  lines  that 
required  analysis  and  identification.  These  conditions 


were  chosen  to  represent  a  range  of  operational  conditions 
from  moderate  to  high  intensity  based  upon  SME  input. 

3.  RESULTS 

Data  collected  from  the  model  fell  into  two  main 
categories,  system  performance  and  estimates  of  operator 
workload.  For  present  purposes  system  performance  data 
will  be  presented  in  terms  of  the  number  of  times  the 
sanitization  process  was  completed  and  the  total  number 
of  contacts  identified  and  logged. 

3.1  System  performance 

The  magnitude  of  the  performance  gain  in  the  sanitization 
process  resulting  from  the  template  assistance  is  shown  in 
Table  1.  These  data  were  the  number  of  complete 
sanitizations  of  all  44  beams  completed  in  the 
approximately  eight  hour  run. 

Run  TGship=25  lines  TGship=100  lines 

Baseline  Assisted  Baseline  Assisted 


1 

4 

16 

1 

8 

2 

4 

17 

1 

8 

3 

4 

17 

1 

7 

4 

4 

17 

1 

10 

5 

4 

18 

1 

8 

6 

4 

18 

1 

8 

7 

4 

17 

1 

8 

8 

4 

17 

1 

8 

9 

4 

18 

1 

7 

10 

4 

17 

1 

8 

Mean 

4 

17.2 

1 

8 

SD 

0 

0.63 

0 

0.81 

Table  1 :  Number  of  complete  sanitizations  completed  for 
baseline  and  assisted  conditions,  where  each  TG  ship  was 
either  25  or  100  lines 

As  can  be  seen  there  is  little  variance  across  runs  and  the 
data  consistently  show  a  4.4  times  increase  in  sanitization 
rate  for  the  low  load  condition  and  a  gain  of  8  times  in 
sanitization  rate  for  the  high  load  condition.  Because, 
only  complete  sanitizations  cycles  were  calculated,  the 
data  for  the  high  load  condition  mean  that  the  operator 
had  completed  one  full  cycle  and  was  in  the  process  of  the 
second  cycle  when  the  run  terminated. 

Given  that  the  assisted  condition  produces  more  efficient 
sanitization  performance,  we  should  expect  to  see  some 
impact  when  the  resulting  residual  spare  capacity  of  the 
sanitizing  operator  is  redirected  to  the  search/id  task.  The 
following  table  addresses  this  issue  and  shows  the  number 
of  contacts  classified  for  each  of  the  test  conditions. 
Clearly,  there  is  a  gain  in  performance  for  both  load 
conditions  and  statistical  analysis  of  these  data  showed 
that  for  both  the  25  line  (t=13.38,  df=18,  p<.01)  and  100 
line  (1=14.47,  df=18,  p<.01)  conditions  there  was  a 


significant  increase  in  the  number  of  contacts  logged 
when  the  task  of  TG  sanitization  was  template  assisted. 

Run  TGship=25  lines  TGship=100  lines 


Baseline 

Assisted 

Baseline 

Assisted 

1 

36 

53 

33 

41 

2 

35 

50 

35 

40 

3 

36 

49 

34 

40 

4 

42 

50 

35 

42 

5 

39 

53 

34 

39 

6 

39 

49 

32 

42 

7 

37 

50 

35 

40 

8 

37 

51 

34 

42 

9 

36 

48 

32 

40 

10 

36 

56 

34 

43 

Mean 

37.3 

50.9 

33.8 

40.9 

Delta 

36.5% 

21% 

SD 

2.11 

2.42 

1.135 

1.29 

Table  2\  Number  of  contacts  correctly  identified  for 
baseline  and  assisted  conditions,  where  each  TG  ship  was 
either  25  or  100  lines. 

3.2  Operator  Workload 

Mean  workload  scores  for  each  operator  on  each  of  the 
workload  dimensions  for  each  of  the  experimental 
conditions  were  computed  based  upon  the  individual 
35225  values  computed  by  the  IPME  model  during  each 
of  the  10  runs.  The  means  of  the  ten  runs  were  then 
calculated  and  the  ensuing  data  are  shown  in  Figure  4. 
Data  are  presented  for  the  two  sanitization  conditions  in 
which  the  number  of  lines  per  TG  ship  was  either  25  or 
100  per  beam. 

Statistical  analysis  of  the  data  was  conducted  using  two- 
factor  (baseline/assisted  and  number  of  lines  per  TG  ship) 
analysis  of  variance  (ANOVA).  Where  necessary, 
supplemental  comparisons  were  made  using  t-tests.  It 
was  decided  that  it  would  not  be  appropriate  to  combine 
all  of  the  individual  workload  measures  into  a  single 
multivariate  analysis  of  variance,  in  view  of  the  fact  that 
the  auditory  workload  scores  showed  opposite  effects  to 
the  other  measures  and  that  the  separate  workload  indices 
are  theoretically  uncorrelated. 
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Figure  4\  Mean  workload  ratings  as  a  function  of  test 
condition 

For  the  both  TG  conditions,  there  seems  to  be  a  small 
trend  for  lower  workload  ratings  in  the  assisted  condition 
for  visual,  cognitive  and  psychomotor  components,  and  a 
reverse  trend  for  the  auditory  workload  component.  The 
effect  of  the  number  of  TG  lines  to  be  analysed  was  not 
consistent.  For  all  workload  measures  except  the 
cognitive  (where  the  reverse  was  true),  workload  was 
slightly  higher  in  the  25  line  condition.  However,  these 
effects  are  quite  small,  typically  of  the  order  of  less  than 
.1  on  the  7-point  workload  scale,  but  are  statistically 
significant  because  of  the  small  variance  between 
simulation  runs.  The  significant  interactions  for  visual  and 
cognitive  workload  scores  reflected  a  larger  effect  of  the 
template  assistance  under  the  100  line  condition, 
compared  with  the  25  line  condition.  This  difference  was 
in  the  opposite  direction,  however,  for  the  psychomotor 
scores. 

4.  DISCUSSION 


assisted  conditions  for  the  actual  sanitization  process  may 
have  been  masked  by  the  use  of  session-wide  average 
ratings,  which  include  all  of  the  other  tasks  executed, 
when  there  was  available  time,  beyond  the  visual  process 
of  making  the  template  match. 

While  the  results  of  the  simulation  clearly  pointed  to  the 
potential  value  of  the  aid,  there  was  some  concern  of  their 
validity  from  two  perspectives.  First,  while  the  model 
function  parameters  for  known  sonar  tasks  were  based 
upon  recommendations  from  an  experienced  SME,  the 
values  used  to  estimate  task  performance  with  the 
template  were  derived  from  an  analysis  and  estimation 
from  published  human  factors  data  for  similar  task 
contexts,  together  with  input  derived  from  the  experience 
of  the  human  factors  team.  Second,  the  values  used  for 
the  workload  for  each  task  were  based  upon  the  IPME 
scale  values,  and  there  may  some  issues  with  how  well 
such  values  generalize  to  other  task  situations. 

Accordingly,  it  was  decided  that  there  should  be  an 
attempt  to  assess  the  validity  of  the  modelling  and 
simulation  results  by  collecting  human  performance  data 
with  real  operators  performing  the  sanitization  task  under 
baseline  and  assisted  conditions.  The  critical  data  would 
comprise  both  performance  effectiveness  (in  terms  of 
throughput  or  sanitization  rate)  as  well  as  perceived 
workload  for  each  task  component. 

Two  problems  immediately  presented  themselves  in 
considering  the  validation  approach.  First,  existing  sonar 
systems  do  not  provide  the  kind  of  flexibility  required  to 
serve  as  an  experimental  testbed.  Second,  access  to 
trained  sonar  operators  in  sufficient  numbers  to  generate  a 
statistically  acceptable  design  was  virtually  impossible. 

The  obvious  solution  therefore  was  to  build  a  simple 
simulation  of  the  sonar  sanitization  task  that  would  allow 
us  to  train  non-Navy  personnel  to  the  required  standard  of 
proficiency  and  then  conduct  the  validation  study  with 
such  personnel. 


The  results  of  the  simulation  show  that  the  impact  of 
adopting  a  smart,  visual  template  to  assist  the  sanitize 
process  has  a  significant  effect  on  the  speed  with  which 
this  process  can  be  executed.  As  a  result,  more 
sanitization  cycles  can  be  performed  in  a  watch  and  there 
is  additional  residual  capacity  created,  such  that  the 
“operator”  performing  the  sanitization  task  is  able  to  work 
on  the  search/id  task  thereby  improving  the  overall  rate  of 
identifications  for  that  task  also. 

The  results  showed  minimal  impact  on  mean  predicted 
operator  workload  across  a  test  run  (i.e.  watch  period) 
resulting  from  the  use  of  the  template  for  the  sanitization 
task.  This  may  be  for  two  reasons.  First,  the  numbers 
entered  into  the  model  may  not  have  been  valid  (being 
simply  based  upon  the  IPME  normative  values).  Second, 
any  small  differences  in  workload  between  baseline  and 


Space  limitations  for  the  present  paper  preclude  the 
possibility  of  describing  the  validation  experiment  in 
sufficient  detail.  However,  the  results  can  be  summarized 
by  stating  that  the  human  performance  data  fell  within  the 
range  predicted  by  the  model  and  that  the  predicted  gain 
in  performance  due  to  the  provision  of  the  decision  aiding 
tool  was  generally  supported  by  the  data  obtained.  In 
contrast,  workload  ratings  generated  in  real  time  by  the 
participants  in  the  study  were  consistently  lower  than 
predicted  by  the  model  for  both  baseline  and  assisted 
conditions. 

Thus,  we  were  able  to  conclude  generally  that  the 
modelling  approach  provided  a  reliable  and  valid  method 
for  estimating  the  effectiveness  of  a  decision  aid  on 
system  performance.  Further,  the  creation  of  a 
comprehensive  baseline  model  will  allow  quantifiable 


estimates  to  be  made  on  the  effectiveness  of  other  future 
system  re-design  options, 
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Annex  A:  Table  showing  workload  descriptors 


Ordinal  rating 
(W/Index) 

Interval 
rating  (VACP) 

Descriptor 

VISUAL 

1 

1 

Register/Detect  (Detect  Occurrence  of  Image) 

2 

3.7 

Discriminate  (Detect  Visual  Difference) 

3 

4 

Inspect/Check  (Discrete  Inspection/Static  Condition) 

4 

5 

Locate/Align  (Selective  Orientation) 

5 

5.1 

Track/Follow  (Maintain  Orientation) 

6 

5.9 

Read  (Symbol) 

7 

7 

Scan/Search/Monitor  (Continuous/Serial  Inspection,  Multiple  Conditions) 

AUDITORY 

1 

1 

Detect/Register  Sound  (Detect  Occurrence  of  Sound) 

2 

2 

Orient  to  Sound  (General  Orientation/Attention) 

3 

4.2 

Orient  to  Sound  (Selective  Orientation/Attention) 

4 

4.3 

Verify  Auditory  Feedback  (Detect  Occurrence  of  Anticipated  Sound) 

5 

4.9 

Interpret  Semantic  Content  (Speech) 

6 

6.6 

Discriminate  Sound  Characteristics  (Detect  Auditory  Differences) 

7 

7.0 

Interpret  Sound  Patterns 

COGNITIVE 

1 

1.0 

Automatic  (Simple  Association) 

2 

1.2 

Alternative  Selection 

3 

3.7 

Sign/Signal  Recognition 

4 

4.6 

Evaluation/Judgment  (Consider  single  Aspect) 

5 

5.3 

Encoding/Decoding,  Recall 

6 

6.8 

Evaluation/Judgment  (Consider  Several  Aspects) 

7 

7.0 

Estimation,  Calculation,  Conversion 

PSYCHOMOTOF 

1 

1 

1.0 

Speech 

2 

2.2 

Discrete  Actuation  (Button,  Toggle,  Trigger) 

3 

2.6 

Continuous  Adjustive  (Flight  Control,  Sensor  Control) 

4 

4.6 

Manipulative 

5 

5.8 

Discrete  Adjustive  (Rotary,  Vertical  Thumbwheel,  Lever  position) 

6 

6.5 

Symbolic  Production  (Writing) 

7 

7 

Serial  Discrete  Manipulation  (Keyboard  Entries) 

